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ABSTRACT 



This paper poses a question of direct relevance for welfare 
administrators, program operators, and policy makers: What management 
practices, program strategies, and local conditions are key to running 
effective welf are-to-work programs? To address this question, the present 
analysis links detailed measures of program characteristics to valid and 
precise estimates of program effects on short-term earnings. Data for the 
analysis are drawn from these three random assignment studies of 
welfare- to-work programs in 59 sites across the United States (with a 
combined total of 69,399 welfare clients): California's Greater Avenues for 
Independence (GAIN) program, Florida's Project Independence (PI), and the 
National Evaluation of Welfare- to-Work Strategies (NEWWS) . The findings 
indicate that, other things being equal, program effects on earnings during 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



the first two years after random assignment are largest when programs 
strongly emphasize employment, provide personalized attention, and do not let 
staff caseloads become large. The paper also finds that short-term impacts on 
earnings are smaller where unemployment is high, management choices and 
practices matter a lot, increased reliance on basic education reduces 
short-run effects, economic environment plays an important role, and program 
effectiveness varies inconsistently with client characteristics. Appendixes 
include 54 references; data tables and figures; program models for GAIN, PI, 
and NEWWS; and descriptions of measuring program performance as program 
effects on client earnings, measuring program characteristics, and testing 
the sensitivity of findings from the impact model. (Contains 54 references.) 

( Author /YLB) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



MDRC Working Papers on Research Methodology 



Modeling the Performance of 
Welfare -to-Work Programs: 

The Effects of Program Management 
and Services, Economic Environment, 
and Client Characteristics 



Howard S. Bloom 
Carolyn J. Hill 



James Riccb 



BEST COPY AVAILABLE 



Manpower Demonstration 
Research Corporation 



U.S. DEPARTMENT OF EDUCATION 

Office of Educational Research and Improvement 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



MDRC 





originating it. 



June 2001 



□ Minor changes have heen made to 
improve reproduction quality. 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



This working paper is part of a new series of publications by MDRC on alternative 
methods of evaluating the implementation and impacts of social programs and policies. 



The research reported in the present paper was funded through a subcontract with the University 
of Chicago from grant #97000617-000 awarded by The Pew Charitable Trusts. The authors are 
at MDRC (Bloom and Riccio) and the University of Chicago (Hill). 

The authors are grateful to Laurence E. Lynn, Jr. of the University of Chicago, for valuable 
counsel and support for the present research agenda. For insights into the programs studied, 
assistance with the data used, and other advice, they thank their colleagues Thomas Brock, 
Stephen Freedman, Gayle Hamilton, James Kemple, Charles Michalopoulos, and Electra Small 
of MDRC, and Alan Orenstein. 

MDRC’s evaluation of the California Greater Avenues for Independence (GAIN) Program was 
funded mainly by the California Department of Social Services (CDSS), with additional support 
from the U.S. Department of Health and Human Services (HHS). 

MDRC’s evaluation of Florida’s Project Independence (PI) Program was funded by Florida’s 
State Department of Health and Rehabilitative Services and with additional support from the 
Ford Foundation and HHS. 

MDRC is conducting the National Evaluation of Welfare-to-Work Strategies with funding from 
HHS under a competitive award, Contract No. HHS- 100-89-0030. HHS is also receiving 
funding for the evaluation from the U.S. Department of Education. The study of one of the sites 
in the evaluation, Riverside County (California), is funded in part by a contract with CDSS 
which, in turn, is receiving additional funds from the California State Job Training Coordinating 
Council, the California Department of Education, HHS, and the Ford Foundation. 

Dissemination of MDRC publications is also supported by the following foundations and 
individuals who help finance MDRC’s public policy outreach and expanding efforts to 
communicate the results and implications of our work to policymakers, practitioners, and others: 
the Ford, Ewing Marion Kauffman, Ambrose Monell, Alcoa, George Gund, Grable, Anheuser- 
Busch, New York Times Company, Heinz Family, and Union Carbide Foundations; and the 
Open Society Institute. 

The findings and conclusions presented in this paper do not necessarily represent the official 
positions or policies of the funders. 



For information about MDRC, see our Web site: www.mdrc.org. 

MDRC® is a registered trademark of the Manpower Demonstration Research Corporation. 
Copyright © 2001 by the Manpower Demonstration Research Corporation. All rights reserved. 




3 



Abstract 



This paper poses a question of direct relevance for welfare administrators, program 
operators, and policy makers: What management practices, program strategies, and local 
conditions are key to running effective welfare-to-work programs? To address this 
question, the present analysis links detailed measures of program characteristics to valid 
and precise estimates of program impacts on short-term earnings. The data for the 
analysis are drawn from three random assignment studies conducted by MDRC of 
welfare-to-work programs in 59 sites across the U.S. (with a combined total of 69,399 
welfare clients): California’s Greater Avenues for Independence (GAIN) program, 
Florida’s Project Independence (PI), and the National Evaluation of Welfare-to-Work 
Strategies (NEWWS). The findings indicate that, other things being equal, program 
impacts on earnings during the first two years after random assignment are largest when 
programs strongly emphasize employment, provide personalized attention, and do not let 
staff caseloads become large. The paper also finds that short-term impacts on earnings are 
smaller where unemployment is high. Future papers will address corresponding questions 
about other labor market and welfare outcomes in the short and longer term. 
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Chapter 1 
Introduction 



The overriding goal of the present study is to help improve the performance of human 
service organizations. The specific objectives that flow from this goal are threefold: (1) to 
provide a conceptual and statistical framework for studying the determinants of the performance 
of such organizations, (2) to illustrate how this framework can be used to study the performance 
of a particular type of human service organization — welfare-to-work programs, and (3) to 
report empirical findings about these programs that are substantively important in their own right. 
To accomplish these goals, the study uses data from randomized experimental evaluations of 
three major welfare-to-work programs that operated during the 1990s: California's Greater 
Avenues for Independence (GAIN) program, Florida’s Project Independence (PI), and the 
National Evaluation of Welfare-to-Work Strategies (NEWWS). 

Specifically, the present study estimates the effects of program management, program 
services, economic environment, and client characteristics on the performance (impacts) of the 
preceding programs in terms of one key outcome — mean client earnings during the first two 
years after random assignment. 1 Findings from the study — based on randomized experiments in 
59 locations with a total of 69,399 program and control group members— suggest that, holding 
other factors constant {ceteris paribus)'. 

Management choices and practices matter a lot. Program impacts increase 
substantially when a program is managed in such a way as to provide a clearer and 
more consistent focus on quick employment and/or greater staff emphasis on 
personalized attention to their clients. Program impacts decrease substantially when 
staff caseloads are large. 

Increased reliance on basic education reduces short-run effects. Offices that 
rely more on basic education have lower impacts on client earnings during the first 
two years after random assignment, other factors equal, compared to offices that 
rely relatively less on basic education services. 

Economic environment plays an important role. Program- induced earnings 
gains are larger in areas with lower unemployment rates. 

Program effectiveness varies inconsistently with client characteristics. No 
clear pattern emerges whereby program impacts are consistently larger or smaller 
for more disadvantaged or less disadvantaged clients. For example, there are no 
statistically significant relationships between program impacts and clients’ age, 
race/ethnicity, or pre-program earnings. And while impacts are larger than average 



1 Future analyses will explore corresponding findings for other labor market and welfare outcomes and over 
a longer period of follow-up. 



for clients with at least a high school degree or GED, they are also larger for clients 
who had been receiving welfare payments consistently for at least one year prior to 
the program, and for those with three or more children. 

To help motivate, present, and explain these findings and how they were derived, the 
paper proceeds as follows. The current chapter provides an overview of the policy context of 
the analysis and introduces the issues to be addressed. Chapter 2 outlines the analytic 
framework used. Chapter 3 describes the settings, the sample, and the measures to which the 
analytic framework was applied. Chapter 4 reports the empirical findings that were obtained 
and Chapter 5 concludes the paper with a summary of its main findings and a discussion of next 
steps for future research. 

1.1 Improving the Performance of Human Service Organizations 2 

Human service organizations can be defined in many ways, but in general, they comprise 
“formal organizations explicitly designed to process and change people” (Hasenfeld and English, 
1974). Such organizations include, among others: schools, employment agencies, welfare 
agencies, correctional institutions, hospitals, and mental health clinics. Although the place of 
human services in modem society is fairly secure, public complaints about their inadequate 
performance and calls for their reform are as old as the institutions themselves. 

Improving the performance of human service organizations is a difficult task, in large 
part because rigorous empirical evidence is sorely lacking on the question of “what works best 
for whom?” If social scientists are to help fill this information gap, they must make considerable 
progress on at least two challenging fronts: (1) improving the measurement of performance, and 
(2) increasing knowledge about what drives performance. Clearly, it is not possible to know 
whether performance has improved if one cannot measure it. Equally clearly, it is very difficult to 
improve performance if one does not know what affects it. 

1.2 Measuring Performance 

Measuring the performance of a human service organization requires a clear 
specification of the organization's intended outcomes and an ability to identify its contribution to 
those outcomes. 

1.2.1 Specifying Intended Organizational Outcomes 

The first requirement — the existence of clearly specified outcome measures — is much 
easier to meet for a profit-making firm than it is for a human service organization, which typically 
is a government agency or a non-profit entity. Among for-profit firms, the primary benchmark of 



2 The remainder of Chapter 1 draws heavily on an earlier paper on this project (see Riccio, Bloom, and Hill, 
2000 ). 



